fix: share one HTTP client per provider and size its connection pool - #1021

Closes #1017.

Outpost built a separate http.Client for every destination and never configured pooling on it, so each inherited Go's default of two idle connections. Above two concurrent deliveries to a destination, reuse collapsed to roughly one new connection per delivery.

What changed

One client per provider. Nothing in the client configuration varies per destination — user agent, proxy settings and the transport wrapper all come from provider-level options fixed at registration — so the client moves from CreatePublisher to the provider constructor (destwebhook, destwebhookstandard, desthookdeck). This also means publisher-cache eviction no longer throws away the connection pool.

The rule the code now documents: only connection-level concerns justify a separate client. If per-destination proxy or client certificates are needed later, transports should be keyed by configuration and shared within a key.

Derived pool sizing (internal/destregistry/connpool.go) — sane defaults scaling with DELIVERY_MAX_CONCURRENCY (call it C):

MaxIdleConns caps only parked (idle) connections, never active ones — exceeding it causes connection churn, not errors. That makes the constants safe to reason about as reuse-rate tuning:

• 32 × C ≈ how many distinct destinations a worker revisits within the 90s IdleConnTimeout at ~3s per delivery — slow destinations are where reuse matters most.
• Floor 512 covers low-concurrency fanout: concurrency bounds simultaneous requests, not distinct hosts touched over time. C=1 at ~100ms per delivery still cycles ~900 destinations per idle window.
• Cap 4096 bounds the parked-FD/memory cost where the reuse hit rate decays — but the cap rises to C, so it never undersizes the pool below the concurrency level.

The hookdeck provider talks to one host, so it gets a depth-only pool.

Why not derive from RLIMIT_NOFILE (an earlier revision did): the limit is per-process and unreliable in containers — the RLIM_INFINITY quirk meant the most generous environments derived the smallest pools — and mq-provider publishers hold per-destination FDs the derivation never accounted for. Since the limit only governs idle connections, tying it to a hard resource ceiling was solving the wrong problem.

No new configuration. These are defaults that scale with the one knob operators already set for delivery throughput. If a workload needs different sizing, an explicit config knob can be exposed later — adding one is backward-compatible; removing one isn't. Meanwhile:

• resolved values logged at startup (delivery_max_idle_conns, delivery_max_idle_conns_per_host)
• new outpost.delivery_connections counter, dimensioned by type and reused — the signal that the ceiling is binding

Tests

internal/destregistry/httpclient_pool_test.go counts TCP connections opened at an httptest server via ConnState, swept across concurrency levels against both fast and slow destinations — the two regimes fail differently (latency vs. ephemeral ports). Connections opened track the concurrency level, not the request count.

A control case runs the identical workload through a stock-default client so the assertion can't pass trivially: at concurrency 32, stock opens ~120 connections for 320 requests, sized opens 32.

destwebhook_connpool_test.go covers the provider half — 8 publishers, 40 requests, still bounded by concurrency.

Behavior changes worth flagging

• An invalid DESTINATIONS_WEBHOOK_PROXY_URL now fails at startup instead of on first publish.
• Idle connections are held on the receiver for up to 90 seconds after a delivery. Those connections were opened anyway; pooling changes whether they're held afterward.
• The pool is shared across tenants. Connections are per-host so there's no correctness concern, but a busy tenant can evict a quiet tenant's idle connections; the per-host limit bounds the blast radius.

Verification

go test -short ./... clean. Pool tests run at -count=15 for flake.

🤖 Generated with Claude Code